Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Falcon3 model support #10864

Merged
merged 1 commit into from
Dec 17, 2024
Merged

Add Falcon3 model support #10864

merged 1 commit into from
Dec 17, 2024

Conversation

mokeddembillel
Copy link
Contributor

Adding Falcon3 model support

@github-actions github-actions bot added the python python script changes label Dec 17, 2024
@Vali-98 Vali-98 mentioned this pull request Dec 17, 2024
Open
@ggerganov ggerganov merged commit 382bc7f into ggerganov:master Dec 17, 2024
51 checks passed
@dranger003 dranger003 mentioned this pull request Dec 17, 2024
Closed
slaren added a commit that referenced this pull request Dec 17, 2024
@slaren
Revert "llama : add Falcon3 support (#10864)"
e10dc00 
This reverts commit 382bc7f.
@slaren slaren mentioned this pull request Dec 17, 2024
Merged
slaren added a commit that referenced this pull request Dec 18, 2024
@slaren
Revert "llama : add Falcon3 support (#10864)" (#10876)
4da69d1 
This reverts commit 382bc7f.
@slaren
Copy link
Collaborator

slaren commented Dec 18, 2024

@mokeddembillel heads up, this has been reverted because the change to convert_hf_to_gguf.py was creating gguf files with broken tokenizers. This will need to be fixed before it can be added again.

@mokeddembillel
Copy link
Contributor Author

@slaren @ggerganov, Thanks for flagging this. working on fixing it right now.

@mokeddembillel mokeddembillel mentioned this pull request Dec 18, 2024
Merged
@mokeddembillel
Copy link
Contributor Author

@slaren @ggerganov Thanks again for flagging this issue.

The issue is that when using meta-llama/Llama-3.1-8B-Instruct the <|begin_of_text|> token is added to every special token when doing token = tokenizer.decode(tokenizer.encode(token))

the screenshot shows before and after token = tokenizer.decode(tokenizer.encode(token))
image

I'm fixing this by adding add_special_tokens=False to tokenizer.encode(). Here is the the result after the fix
image

to be extra safe, we will use token = tokenizer.decode(tokenizer.encode(token)) only if len(token) == 1 so that still fix this issue when \n is econded as Ċ

Generation before the fix:

Prompt: Once upon a time in a land far away,
there was a kingdom ruled by a wise and just king. The kingdom was known for its beauty and prosperity, and the people lived in peace and harmony.ĊĊOne day, a terrible drought struck the land, and the crops began to wither and die. The king, worried about the well-being of his people, called upon his wise council to find a solution. The council, after much deliberation, decided to send a group of brave knights to search for a magical spring that was said to have the power to bring rain to the kingdom.

Generation after the fix:

Prompt: Once upon a time in a land far away,
there was a kingdom ruled by a wise and just king. The kingdom was known for its beauty and prosperity, and the people lived in peace and harmony.

One day, a terrible drought struck the land, and the crops began to wither and die. The king, worried about the well-being of his people, called upon his wise council to find a solution. The council, after much deliberation, decided to send a group of brave knights to search for a magical spring that was said to have the power to bring rain to the kingdom.

Created new PR with the Fix #10883

arthw pushed a commit to arthw/llama.cpp that referenced this pull request Dec 20, 2024
@mokeddembillel @arthw
llama : add Falcon3 support (ggerganov#10864)
6a27965
arthw pushed a commit to arthw/llama.cpp that referenced this pull request Dec 20, 2024
@slaren @arthw
Revert "llama : add Falcon3 support (ggerganov#10864)" (ggerganov#10876)
5b74740 
This reverts commit 382bc7f.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ggerganov ggerganov ggerganov approved these changes

Assignees
No one assigned
Labels
python python script changes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@mokeddembillel @slaren @ggerganov